This invention relates to methods of estimating the subjective quality of a multimedia communications system or distributed applications software system in which the quality of the received audio, voice, video or the response time of the software system may be degraded by time varying impairments and in which it is accordingly desirable to have some means of measuring or estimating the subjective or perceptual quality of the decoded audio, voice, video or software system.
It is desirable to have some means of measuring the quality of a distributed system in order that it can be verified that the system is operating correctly and that user expectations are being met. In the case of a speech transmission system such as a telephone network this measurement is sometimes made subjectively by using a test group of people or test surveys. The results of a subjective voice quality test are usually given as a Mean Opinion Score (MOS). As it is generally impractical to conduct subjective tests during the normal operation of a speech transmission system objective testing methods have been developed. These use measured statistics related to the speech transmission system to predict a subjective performance level and typically determine an R rating factor.
Modern communications systems often used some form of shared transmission medium, for example wireless, local or wide area packet switched network. This increases the prevalence of time varying transmission impairments unlike the more constant and predictable impairments characteristic of older circuit switched networks.
It has been found during subjective testing that bursts of noise or distortion occurring late in a call have greater effect than the same bursts occurring early in the call, as illustrated in FIG. 3. This effect is thought to be due to the operation of human short-term memory which retains information on the most recent events and remembers detailed information on heard events for typically two minutes. If a series of bursts of noise or distortion occur during a telephone conversation then according to the known and accepted psychological theories of short term memory and recency the user would tend to remember the last burst and have a general recollection of the call quality up to that point. Experiments by Boneau and Daily (Psychological Association of America, 1995) indicate that the short-term memory loss process is exponential in nature.
Current systems for estimating voice quality do not consider the effects of the temporal location of impairments on the user and hence do not give an accurate representation of subjective voice quality. The present invention relates to methods for estimating subjective voice quality that consider temporal effects and hence can provide greater accuracy.
Distributed applications software can exhibit similar problems. Network or server loading may vary with time, which results in a user experiencing variable response time. It is expected that detailed subjective testing of user satisfaction would reveal similar characteristics to those found in testing telephone networks as the same basic memory processes are at work. The present invention provides a new approach for estimating the subjective quality of distributed applications software.